Audio–Visual Segmentation

We propose to explore a new problem called audio-visual segmentation (AVS), in which the goal is output pixel-level map of object(s) that produce sound at time image frame. To facilitate this research, we construct first benchmark (AVSBench), providing pixel-wise annotations for sounding objects audible videos. Two settings are studied with benchmark: 1) semi-supervised single source and 2) fully-supervised multiple sources. deal AVS problem, method uses temporal interaction module inject audio semantics as guidance visual process. also design regularization loss encourage mapping during training. Quantitative qualitative experiments on AVSBench compare our approach several existing methods from related tasks, demonstrating proposed promising building bridge between semantics. Code available https://github.com/OpenNLPLab/AVSBench .